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to explore the use of simulation technologies for enhancing team training and evaluation. The TRC has adopted theoretical 
frameworks for evaluating training readiness and efficacy, and is using these frameworks to guide a systematic reconfiguration of 
the infrastructure supporting healthcare teams training and research initiatives at NMCP. 


1.0 INTRODUCTION 

Designated a Department of Defense Team 
Resource Center (TRC) in 2008, Naval 
Medical Center Portsmouth (NMCP) 
currently hosts a tri-service healthcare 
teams training course three times annually. 
The course consists of didactic learning 
coupled with simulation exercises to provide 
an interactive educational experience for 
healthcare professionals. 

Simulated cases are developed to reinforce 
specific teamwork skills and behaviors, and 
incorporate a variety of technologies 
including standardized patients, manikins, 
and virtual reality. The course is also the 
foundation of a research program exploring 
the use of modeling and simulation to 
enhance teamwork training in healthcare. 

The TRC has adopted a theoretical 
framework for evaluating training efficacy 
based on Kirkpatrick’s training outcomes 
model [1], and has used this framework to 
guide a systematic reconfiguration of the 
infrastructure supporting healthcare teams 
training and research initiatives at NMCP. 

1.1 Healthcare Teams Training 

Teamwork and communication failures are 
the leading cause of adverse patient 


outcomes [2], [3]. These failures represent a 
gap in functional teamwork behaviors which 
has been addressed in a number of other 
teamwork-intensive industries (e.g., 
aviation) through the introduction of 
carefully designed team skills training 
programs [4], [5], 

Teamwork has also been empirically linked 
to clinical patient outcomes in the 
healthcare domain [3], yet evidence 
suggests that a significant number of health 
care providers hold misconceptions about 
the nature and efficacy of teamwork in their 
own units [6}. Several teamwork (non- 
citnical) skills training programs have 
recently been tailored to the healthcare 
domain; one such program 
(TeamSTEPPS™) is conducted periodically 
at NMCP. 

TeamSTEPPS is a teamwork training 
system that was developed by the U S. 
Department of Defense in partnership with 
the Agency for Healthcare Research and 
Quality [7], aiming to instill positive 
teamwork behaviors in healthcare 
professionals by emphasizing key tenets 
adapted from aviation’s Crew Resource 
Management training system. These include 
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communication, team structure, leadership, 
mutual support, and situation monitoring. 

The NMCP TeamSTEPPS program is 
designed to provide students with the skills 
and tools necessary for effective teamwork, 
as well as hands-on skills training via 
simulation. The key assumptions are that 
critical teamwork skills are reinforced during 
the training program and that patient 
outcomes will improve as a result of these 
skills being transferred to the work 
environment. 

However, recent research on healthcare 
team training efficacy has yielded mixed 
results [8], [4], [5], One possible reason for 
this is the complexity of linking team 
performance characteristics to measurable 
outcomes. Few healthcare team training 
initiatives currently implement a 
comprehensive evaluation protocol, thus 
failing to demonstrate the achievement of 
intermediate training objectives. We have 
developed a multi-level assessment 
protocol for healthcare team training 
outcomes, which includes behavioral 
observation and analyses stemming from 
training scenarios conducted at the NMCP 
Healthcare Simulation Center. 

1.2 Simulation as a Tool for 
Healthcare Teams Training 

Simulation and teamwork are two relatively 
novel aspects of healthcare training that 
have only recently begun to receive 
significant attention. Using a simulated 
operating theater to examine surgical skill 
acquisition and maintenance overtime, 
Moorthy et ai. [9] discovered that 
communication skills (unlike technical skills) 
do not develop naturally as a result of 
increased job experience. Rather, these 
skills must be consciously trained and 
reinforced. Further, the ability of healthcare 
professionals to accurately and reliably 
assess their own non-technical performance 
is not sufficient to promote self- regulation 
and skill acquisition [10], 


Effective teamwork is critical for patient 
safety, yet becoming an expert team 
member requires practice. Evidence is 
beginning to emerge in support of 
simulation as an ideal tool for healthcare 
teams training [11], [4], [12]. At NMCP, we 
have been able to successfully integrate 
simulation into our healthcare teams training 
program with the goal of enhancing 
teamwork skill acquisition through hands-on 
practice. 

1.3 Early Training Infrastructure and 
Resource Availability at NMCP 

1.3.1 Audio/Video Data Support 

The NMCP Healthcare Simulation Center is 
equipped with a network of ceiling-mounted 
video cameras and microphones which feed 
into a central control area for programming 
camera angles, recoding and storing 
audio/video data, and rendering hard-copy 
discs. This network covers the entire center, 
with several cameras and microphones 
dedicated to each training room throughout 
the center. For training purposes, it is often 
desirable to play back the video feed so that 
participants have a chance to review 
performance and debrief individually or with 
instructors. 

1.3.2 Patient Simulators 

The Simulation Center houses a number of 
manikin simulators, including the Laerdal 
SimMan 3G, Laerdal SimBaby NEWBIE, 
and METI iStan. Training scenarios for the 
manikins can be created and delivered 
using either PC or Mac systems, depending 
on user needs. A larger number of part-task 
skills and box trainers are also available, 
covering a range of tasks such as IV line 
insertion, orthroscopy, central line 
placement, airway management, suturing, 
lumbar puncture, and more. 

1.3.3 Virtual Reality 

To support the acquisition and maintenance 
of psychomotor skills for minimally invasive 
surgery (MIS), the Simulation Center offers 
a variety of virtual reality (VR) trainers. 
These VR trainers can be used practice 
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upper and lower Gl respiratory scope 
insertion, colonoscopy, hernia repair, 
laparoscopic cholecystectomy, and more. 
Each of the VR systems provides students 
with a set of physical instruments that must 
be manipulated in order to affect some 
physiological change on a virtual patient. 
These changes are fed back to the student 
through display of realistic sights and 
sounds (delivered through digital monitors) 
and touch feedback (delivered through the 
instruments themselves). 

1.3.4 Standardized Patients 

The Team Resource Center at NMCP 
currently partners with the Theresa A. 
Thomas Professional Skills Teaching and 
Assessment Center at the Eastern Virginia 
Medical School to provide standardized 
patients in support of our training courses. 
Standardized patients are highly-trained 
actors who present symptoms of illness and 
then assess healthcare professionals 1 
diagnostic and interpersonal skills during 
face-to-face interactions. The standardized 
patients are carefully integrated into training 
scenarios to produce a realistic social 
context for students to practice within. 

1.4 Performance Assessment: 
Kirkpatrick Training Model 

The NMCP Healthcare Simulation Center 
offers a range of simulation technologies to 
support team training. However, technology 
alone is not the key to training success. 
Simulation must be part of a larger training 
process, including a well-designed 
curriculum and evaluation protocol. The 
latter is perhaps more often neglected than 
the former [4], [5] TeamSTEPPS students 
at NMCP are evaluated throughout the 
course along four levels of measurable 
outcome: reaction, learning, behavior, and 
results. These levels are based on 
Kirkpatrick’s [1] model of training outcomes 
assessment. 

1.4.1 Level 1: Reaction 

Reaction-level feedback reflects the degree 
to which the training course and its content 
are valued by the students. This generally 


consists of asking students to complete a 
short pencil-and-paper feedback survey at 
the end of the course. Reaction-level data 
also help course administrators identify 
program strengths and opportunities for 
improvement. 

1.4.2 Level 2: Learning 

Learning-level feedback represents the 
degree to which relevant student attitudes 
and knowledge are positively impacted by 
participation in the course. Instruments such 
as the Team Attitudes Questionnaire [13] 
and modified TeamSTEPPS knowledge 
assessment instruments [7] are 
administered before training to establish 
baseline metrics. Upon completion of the 
course, students are asked to complete 
these assessments a second time to 
establish a comparison metric. The 
instruments may be continually 
administered over time to determine 
whether teamwork attitudes and knowledge 
have been sustained. 

1.4.3 Level 3: Behavior 

Behavioral outcomes reflect the degree to 
which core TeamSTEPPS tools and 
strategies have been successfully 
implemented in hospital units. To generate 
this form of data, trained observers spend 
time in the units monitoring and recording 
team activity using behavioral checklists. 
Behavior-level feedback is also generated 
during the training course, when students 
are asked to apply TeamSTEPPS concepts 
to resolve simulated case scenarios. These 
simulation sessions are audio/video 
recorded, and performance is critiqued 
during a post-scenario debrief, 

1 .4.4 Level 4: Results 

Unit-specific metrics are maintained on a 
unit-by-unit basis and are analyzed 
periodically by the TRC. These metrics may 
reflect patient outcome data, procedural 
checklists, brief/debrief content analysis, 
and a number of other teamwork-related 
evaluations. Results-level outcomes reflect 
the organizational impact of the 
TeamSTEPPS training program over time. 
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1.5 TRC Performance and 
Assessment Needs 

The first two outcome levels are assessed 
with pencil-and-paper survey instruments 
designed to record students’ perceptions, 
knowledge and attitudes toward team 
training. Behavioral outcomes (Level 3) 
involve demonstration of acquired skill 
through hands-on TeamSTEPPS 
implementation. This is unlikely to occur in 
the work setting unless students are 
provided sufficient practice and feedback 
during training. 

Carefully designed simulation scenarios 
allow students to practice using 
TeamSTEPPS skills and strategies in a safe 
learning environment and to receive 
feedback from colleagues and instructors so 
that these skills can be reinforced. However, 
conducting team training scenarios and 
video debrief sessions for TeamSTEPPS 
was not originally possible due to training 
infrastructure incompatibility. 

Early in the training program it was 
determined that the Simulation Center’s 
audio/video network was not designed to 
support teamwork debriefing. Rather, the 
ceiling-mounted video cameras and audio 
devices were installed to provide top-down, 
patient-centered perspectives for evaluating 
clinical proficiency. The cameras and 
microphones themselves produced low- 
grade surveillance quality sound and 
imaging. Further, the computer system 
dedicated to rendering hard-copy discs of 
the audio/video data for the purposes of 
analysis and debriefing required several 
hours to process, making immediate training 
debriefs impractical. 

In addition to training debriefs, high-quality 
audio/video data were necessary to train 
unit and ward observers, to analyze 
effectiveness of training scenarios, and to 
demonstrate TeamSTEPPS skill 
improvement over a number of trials. Aside 
from the training center’s infrastructure 
incompatibility for TeamSTEPPS, it was 
also determined that our conventional 


teamwork simulation scenarios were not 
producing the desired learning effects. 

Our initial approach to scenario 
development was to embed specific 
TeamSTEPPS learning objectives into a 
series of patient-centered clinical scenarios, 
with roles and learning opportunities 
available for all members of the healthcare 
team. The goal was to provide for clinical 
fidelity at the highest possible level, thus 
allowing students to focus on improving 
teamwork rather than becoming distracted 
by an unfamiliar technical context (e.g., lack 
of functioning anesthesia machine, absence 
of an attending physician, or varying the 
point at which an official timeout is 
conducted before surgery). However, it 
quickly became evident that our strong 
emphasis on clinical detail was 
counterintuitive to our goals of delivering 
quality non-clinical training scenarios. 

The majority of TeamSTEPPS students 
were from outside NMCP and were 
registered to attend by TRICARE 
Management Activity (TMA). Frequently we 
did not know the students' background 
information (i.e., job role, specialty, training 
needs, etc.) in advance. As a result, 
simulation scenarios targeting students with 
specific job roles and skill requirements 
proved to be too inflexible and were difficult 
to manage from an administrative 
perspective. 

We have also observed that as clinical 
fidelity of a given scenario increases, so too 
does student criticism of minor 
inconsistencies between the scenario and 
their own unique work environments. This 
pattern of student reaction to clinical fidelity 
in training scenarios resembles the 
"uncanny valley” phenomenon I14J, in which 
greater fidelity can be associated with 
increased criticism of observable 
discrepancies under certain conditions. 

One potential solution to our scenario 
development process was to de-emphasize 
the clinical nature of scenarios in favor of a 
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stronger non-clinical focus by expanding the 
training roles of our standardized patients. 

2.0 INFRASTRUCTURE EVOLUTION 

In order to maximize TeamSTEPPS training 
efficacy, a number of modifications and 
upgrades to the system’s infrastructure 
were required. First, the existing audio/video 
system was upgraded to support team 
training. Second, the traditional model for 
healthcare simulation training scenarios was 
modified to de-emphasize clinical detail and 
focus on the non-clinical context of team 
training content. 

2.1 Audio/Video System Upgrades 

To enhance data collection and training 
debriefs, all ceiling cameras and 
microphones in the Simulation Center were 
upgraded to high-quality resolution systems. 
Additionally, a number of wall-mounted 
cameras were installed for the purpose of 
capturing team performance using eye-level 
panning. The wall-mounted cameras 
provide screen coverage of team 
performance otherwise unattainable by 
ceiling-mount cameras. The visualization 
control center was upgraded to include new 
monitors and selector switches for 
improving coordination among the cameras 
and microphones. A shoulder camera was 
purchased and incorporated into the data 
collection network, and a new dedicated 
computer system was installed for rendering 
hard-copy discs in minutes rather than 
hours (allowing for immediate debriefs). 

2.2 Training Scenario Modifications 

We expanded the utilization of our 
standardized patients by devising a new 
form of simulation training scenario. Rather 
than focusing on increased clinical fidelity 
for patient-centered, student-driven 
scenarios, we decided to pilot a series of 
scenarios which de-emphasize clinical 
details and focus instead on providing a 
high-fidelity social context in which students 
can practice non-clinical TeamSTEPPS 
skills. Our new scenarios each constitute a 
carefully scripted sequence of events which 


unfold in a generic healthcare setting, but 
do not involve clinical activities and are not 
patient-centric. This ensures that any 
student, regardless of background or job 
role, may freely participate in any of our 
scenarios. 

The scripted scenario is carried out by a 
team of trained actors while students 
observe nearby. The team of actors 
engages in a sequence of social 
interactions with each other while the 
scenario unfolds, and some of the 
interactions are scripted to reflect sub- 
optimal teamwork decisions and behaviors. 
At various points, individual students are 
asked to step into the scenario as a 
participant and attempt to successfully 
resolve an escalating situation by drawing 
on their TeamSTEPPS training. 

Each student is given multiple opportunities 
to engage the actors throughout the 
scenario. As they do so, they will receive 
realistic, immediate feedback from the 
actors in the form of improvised reactions. 
For example, a student who attempts to 
address an actor-physician’s unprofessional 
behavior may receive a passive, hostile, or 
defensive response from the actor. This 
“interactive theater” simulation provides 
multiple opportunities for students to 
practice teamwork skills throughout and 
also supports continual student 
discussion/debriefing as part of the learning 
exercise. 

3.0 IMPACT OF SYSTEMS 
EVOLUTION 

3.1 Audio/Video Capabilities 

As a result of the Simulation Center 
audio/video system upgrades, we are now 
able to record complete scenario sessions 
from multiple viewpoints and perspectives. 
We are also able to capture the entire 
student group in a single frame and identify 
sources of communication (including non- 
verbal) as events unfold. Complete 
audio/video integration and hard-copy disc 
transfer is possible within a matter of 
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minutes, which permits relatively immediate 
video debriefs for students. This not only 
improves the quality of the training 
experience for learners, but also provides 
administrators with a record of how well the 
scenario functioned as a learning exercise. 
Additionally, recorded scenarios serve as 
training material for volunteer unit observers 
learning how to use TeamSTEPPS 
behavioral observation tools in the hospital. 

3.2 Actor-driven Scenarios 

Two non-clinical, actor-driven simulation 
scenarios were piloted in July 2010 at 
NCMP. Overall, the new format for 
TeamSTEPPS simulation training was 
considered a success. Student reactions to 
the actor-driven scenarios were positive. 
Because the emphasis was placed on social 
rather than clinical events, each scenario 
provided students multiple opportunities to 
engage without requiring a specific degree 
of clinical training or job role. This added 
flexibility gives us the ability to include 
students from a variety of backgrounds, 
including hospital administrative staff 
without any clinical training. 

The new format also resulted in a greater 
amount of Team STEP PS- related dialogue 
during post-scenario debriefs, whereas 
clinical scenarios tend to be dominated by 
discussion of clinical activity and 
performance. Actor-driven event scripts 
guaranteed that the scenario would unfold 
in a manner consistent with our established 
learning objectives, whereas previous 
student-driven scenarios required constant 
interjection and management from staff 
scenario “directors.” 

4.0 FUTURE DIRECTION/GOALS 

The NMCP Team Resource Center has 
adopted a theoretical framework for 
evaluating healthcare teams training 
efficacy based on Kirkpatrick’s model of 
training outcomes assessment [1], and is 
using this framework to guide a systematic 
reconfiguration of the infrastructure 
supporting training and research initiatives 
at NMCP. 


Early reaction- level feedback suggested 
that as efforts were made to increase the 
clinical fidelity of training scenarios, 
students were becoming more critical of 
minor inconsistencies within the clinical 
context. It was not our intention to 
emphasize technical proficiency with these 
scenarios, yet providing a practice 
environment with high clinical fidelity 
resulted in a preoccupation with clinical 
performance by our students. Therefore, the 
administrative team found it necessary to 
step back and reconsider the methodologies 
being used to develop TeamSTEPPS 
training scenarios. 

Drawing on Benner’s stages of clinical 
competence [15], our team began to 
reassess the students’ readiness and 
progression with regard to TeamSTEPPS 
skill development. Benner’s theory is based 
on the Dreyfus model of skill acquisition 
[16], which delineates five stages of 
increasing skill: novice, advanced beginner, 
competent, proficient, and expert. Our 
students were considered to be clinically 
proficient {to expert) within their own 
respective disciplines, yet were advanced 
beginners at best within the areas of 
communication and teamwork. Our primary 
goal was to facilitate student transformation 
from an advanced beginner in 
TeamSTEPPS to functional competence by 
the end of the 2,5-day training course. 

Specifically, the identified learning 
objectives for course completion were a.) to 
demonstrate competence in the use of 
TeamSTEPPS strategies and techniques, 
b.) to be able to initiate TeamSTEPPS 
activities upon returning to the students’ 
parent command, and c.) to recognize that 
developing the skills required to become a 
proficient TeamSTEPPS practitioner would 
require continued use of the strategies and 
techniques learned during the course. The 
distinction between the advanced beginner 
and competent skill levels was the guiding 
force behind our shift to a training scenario 
model emphasizing the social rather than 
clinical context. 
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The TRC’s new model of actor-driven 
training scenarios reflects efforts to help 
students achieve Team STEP PS 
competence and to capitalize on 
Kirkpatrick's Level 3 (Behavior) training 
outcome [1], The goal was to provide 
students with ample opportunities to apply 
TeamSTEPPS skills and strategies in a safe 
educational environment where immediate 
feedback could facilitate learning. 
Standardized patients are capable of 
providing students with two forms of 
feedback during these training scenarios: 
real-time improvisational feedback and post- 
scenario debrief feedback. The former 
constitutes a variety of realistic actor 
responses directed toward the students as 
they practice resolving teamwork issues 
throughout each scenario. The latter is an 
overall performance critique presented by 
the actor after the scenario has ended. 

Standardized patient actors have been 
shown to be a reliable and valid means of 
assessing healthcare professionals' non- 
technical skills (17], (18]. The TRC is 
currently developing a standardized protocol 
for assessing students’ TeamSTEPPS 
performance during simulated scenarios; 
the results of these assessments will serve 
as discussion during post-scenario debrief 
sessions. However, as with any formal 
assessment protocol, it will be critical to 
ensure that our assessments are not 
influenced by evaluator bias. 

As we develop a standardized protocol for 
TeamSTEPPS skills assessment, we will 
examine the degree to which evaluator bias 
impacts ratings of student performance [18], 
[19]. Inconsistencies in actors’ role 
portrayal, improvised feedback, or scoring 
could be the result of unique biases (e.g., 
gender, age) attributable to the actor- 
evaluators. One methodology that has been 
developed to assess standardized patient 
bias and establish inter-rater reliability is the 
use of “standardized examinees” [19]. 
Standardized examinees are individuals 
trained to a specific level of proficiency, 
after which they are subjected to 


assessment by a number of standardized 
patients. Inter-rater reliability can then be 
established and potential biases explored 
through the analysis of ratings provided by 
the various standardized patients. 

The most important goals of the TRC 
moving forward are to continue stripping 
away the veneer of expertise that comes 
from students’ confidence in their respective 
technical abilities and to reinforce the notion 
that a significant amount of learning still lies 
ahead for those who would develop 
teamwork expertise as well. It is our 
objective to provide meaningful learning 
experiences for students so that they 
complete the TeamSTEPPS course with the 
competence to implement TeamSTEPPS 
strategies and the motivation required to 
transform their competence into expertise. 
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Theoretically- Driven Infrastructure for 
Supporting Healthcare Teams Training 
at a Military Treatment Facility 




Naval Medical Center Portsmouth (VA) 


First and Finest 
* Construction began 1827 


* Completed in 1830 

O nirloct" hncnifal in Mav/w cwchom 
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Naval Medical Center Portsmouth (VA) 


Residency programs in 13 specialties 

300+ clinical and 140 specialty exam rooms, 17 
operating rooms 

Designated DoD Team Resource Center (TRC) in 
2008 

DoD/AHRQ Team STEP PS™ program 

- 3 tri-service TtT courses annually 

- Command Orientation classes monthly 



Naval Medical Center Portsmouth, VA 


Healthcare Teams Training 

Teamwork/communication breakdowns 

- Root cause in most sentinel events 

- Effective teamwork critical for patient safety 

- Behaviors do not develop naturally d+i 

• Consciously trained 

• Reinforced 

• Mastered over time 

- Successfully addressed in other high-risk domains 

• Crew Resource Management (aviation) 

• TeamSTEPPS™ (healthcare) 

- Department of Defense 

- Agency for Healthcare Research & Quality 

- Jury still out on healthcare teams training £ 5 ' 7 £ 



TeamSTEPPS from CRM 


• TeamSTEPPS core competencies 



- Team Structure 

- Leadership 

- Situation Monitoring 

- Mutual Support 

- Communication 
• TRC program 

- Prepare to lead by example 

- Support change teams 

- Study modeling & simulation technologies 
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Healthcare Simulation Center 


Training/Research Assets 

- Didactic training facilities 

- Audio/ Video capture system 

- Manikin 

- Virtual Reality 

- Standardized Patients 
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Simulation Center Infrastructure Evolution 




Kirkpatrick ^Training Outcomes Model 


Initial 

State 

Current 

State 

Ceiling 
mount only 

Ceiling, wall 
mounts 

Surveillance 
quality video 

High 

resolution 

video 

Delayed 

debriefs 

immediate 

debriefs 



Simulation Training Scenario Evolution 


• Original model: Student-driven scenario 

- Identify TeamSTEPPS training objectives 

- Outline clinical context & key events 

- Set the stage 

- Direct the scenario 
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Simulation Training Scenario Evolution 



• Challenges 

- Specific role requirements (anesthesia) 

- Specialty areas (ED, Dental) 


- Content validity (conflict resolution) 

- Information availability -> flexibility of scenarios 



Simulation Training Scenario Evolution 



• Current Model: Actor-driven scenario 



- Identify Team STEPPS training objectives 

- Script key events, clinical or non-clinical 

- Set the stage 

- Students engage actors intermittently 

- Debrief 
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Simulation Training Scenario Evolution 


• Outcomes 


fjy 



- Flexible scenarios 

• Students practice skills regardless 
of background and experience 

• No specific team structure required 

- Improved scenario control 

- Increased opportunities for 
feedback 

• Actor & Peer feedback: 

- Immediate 

- General 




- Greater content validity 

- Social over clinical fidelity 

• Clinical experts vs. social novices P- 10 1 



The Actor-driven Scenario 



The Dreyfus Model 



"5 

I 

LA 



Expert i Needs to expand knowledge and experience 


Proficient I Needs unhindered practice and "the big picture* 



Competent : Needs real world exposure. 



Advanced Beginner : Needs simple, controlled simulations. 


Novice : Needs recipes, monitoring and first successes. 


12 
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